Visualize monthly changes in Hirakund reservoir using video¶

This Notebook has been adapted from an training authored by Esri, found here and offered under Apache License (2.0).

Copy of original license

Original Copywrite Notice:

Licensing¶

Copyright 2018-2022 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

Significant changes made by NCAR:

  1. Changed the method for detecting and installing libraries (pillow, imageio).
  2. Connecting to the 'home' environment instead of the Esri sandbox.
  3. Changes to collection and create_movie functions to accept an output directory
  4. Displaying the resulting GIF using IPython display Image function instead of hard-coded.
  5. Set the extent of the desired image using India CWC reservoir polygon data.

Table of Contents¶

  • Introduction
  • Necessary Imports
  • Connect to your GIS
  • Get the data for analysis
  • Function to create collection of images with desired time intervals
  • Make video from image collection
  • Conclusion

Introduction¶

The World is changing daily. Major changes like shrinking of lakes, river path shifts, construction of megastructures can be seen directly from the satellite images. This notebook creates a movie to visualize monthly changes in Hirakund reservoir, Odisha.

First, let's import all the necessary libraries and connect to our GIS via an existing profile, e.g. gis = GIS("Pro")

Necessary Imports¶

In [202]:
# Find the directory of the current project and add to PATH
import sys, os, arcpy
home_folder = arcpy.mp.ArcGISProject("current").homeFolder
sys.path.insert(0, home_folder)
os.chdir(home_folder)

# The 00_environment_setup notebook contains libraries and other things common to all the notebooks (e.g. file paths)
%run "00_environment_setup.ipynb"

# Additional libraries
from platform import system
from arcgis.raster.functions import apply

# Because we will later import Image from PIL, we need to import IPython.display.Image as something else
from IPython.display import Image as dImage

Active Portal in ArcGIS Pro
Logged in as ksampson
Current conda environment:
	arcgispro-py3-clone C:\Users\ksampson\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone
Found input data directory: C:\Users\ksampson\Documents\GitHub\GloFAS_Q2Q_Bias_Correction_and_Verification\data\input
Completed importing and/or installing libraries in 4.65 seconds.

Note: to run this sample, you need a few extra libraries in your conda environment. If you don't have the libraries, install them by running the following commands from cmd.exe or your shell

conda install -c anaconda pillow
conda install -c conda-forge imageio
In [203]:
try:
    from PIL import Image, ImageFont, ImageDraw
except ImportError:
    print('Could not import pillow library. Installing using conda.')
    #!conda install -c anaconda pillow -y
    !conda install --freeze-installed esri::numpy=1.20.* anaconda::pillow -y
    from PIL import Image, ImageFont, ImageDraw
    
try:
    import imageio
except ImportError:
    print('Could not import imageio library. Installing using conda.')
    #!conda install -c conda-forge imageio -y
    !conda install --freeze-installed esri::numpy=1.20.* conda-forge::imageio -y
    import imageio

Could not import imageio library. Installing using conda.
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\ksampson\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone

  added / updated specs:
    - imageio


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           1 KB  conda-forge
    cffi-1.16.0                |   py39ha55989b_0         231 KB  conda-forge
    cryptography-41.0.4        |   py39hb6bd5e6_0         1.0 MB  conda-forge
    freetype-2.12.1            |       hdaf720e_2         498 KB  conda-forge
    imageio-2.31.5             |     pyh8c1a49c_0         284 KB  conda-forge
    libsqlite-3.43.2           |       hcfcfb64_0         827 KB  conda-forge
    m2w64-gcc-libgfortran-5.3.0|                6         342 KB  conda-forge
    m2w64-gcc-libs-5.3.0       |                7         520 KB  conda-forge
    m2w64-gcc-libs-core-5.3.0  |                7         214 KB  conda-forge
    m2w64-gmp-6.1.0            |                2         726 KB  conda-forge
    m2w64-libwinpthread-git-5.0.0.4634.697f757|                2          31 KB  conda-forge
    msys2-conda-epoch-20160418 |                1           3 KB  conda-forge
    openssl-3.1.3              |       hcfcfb64_0         7.1 MB  conda-forge
    pip-23.3                   |     pyhd8ed1ab_0         1.3 MB  conda-forge
    setuptools-68.2.2          |     pyhd8ed1ab_0         454 KB  conda-forge
    sqlite-3.43.2              |       hcfcfb64_0         830 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        14.3 MB

The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/win-64::ca-certificates-2023.7.22-h56e8100_0
  imageio            conda-forge/noarch::imageio-2.31.5-pyh8c1a49c_0
  libblas            conda-forge/win-64::libblas-3.9.0-1_h8933c1f_netlib
  libcblas           conda-forge/win-64::libcblas-3.9.0-5_hd5c7e75_netlib
  libiconv           conda-forge/win-64::libiconv-1.17-h8ffe710_0
  liblapack          conda-forge/win-64::liblapack-3.9.0-5_hd5c7e75_netlib
  libsqlite          conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
  m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
  m2w64-gcc-libs     conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
  m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
  m2w64-gmp          conda-forge/win-64::m2w64-gmp-6.1.0-2
  m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
  msys2-conda-epoch  conda-forge/win-64::msys2-conda-epoch-20160418-1
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.36.32532-hdcecf7f_17

The following packages will be UPDATED:

  cffi                pkgs/main::cffi-1.15.1-py39h2bbff1b_3 --> conda-forge::cffi-1.16.0-py39ha55989b_0
  cryptography             esri::cryptography-41.0.3-py39_2 --> conda-forge::cryptography-41.0.4-py39hb6bd5e6_0
  jpeg                                      esri::jpeg-9e-0 --> conda-forge::jpeg-9e-hcfcfb64_3
  libdeflate                                 1.8-h2bbff1b_5 --> 1.17-h2bbff1b_1
  libpng                pkgs/main::libpng-1.6.37-h2a8f88b_0 --> conda-forge::libpng-1.6.39-h19919ed_0
  libtiff                                           4.5.0-2 --> 4.5.1-0
  lz4-c                   pkgs/main::lz4-c-1.9.3-h2bbff1b_1 --> conda-forge::lz4-c-1.9.4-hcfcfb64_0
  numpy                           esri::numpy-1.20.1-py39_0 --> conda-forge::numpy-1.20.3-py39h6635163_0
  numpy-base                                  1.20.1-py39_0 --> 1.24.3-py39_1
  olefile              pkgs/main::olefile-0.46-pyhd3eb1b0_0 --> conda-forge::olefile-0.46-pyh9f0ad1d_1
  openssl                            esri::openssl-3.0.10-0 --> conda-forge::openssl-3.1.3-hcfcfb64_0
  pillow                                       9.5.0-py39_0 --> 9.5.0-py39_1
  pyopenssl          pkgs/main/win-64::pyopenssl-23.2.0-py~ --> conda-forge/noarch::pyopenssl-23.2.0-pyhd8ed1ab_1
  setuptools          esri/win-64::setuptools-67.7.2-py39_0 --> conda-forge/noarch::setuptools-68.2.2-pyhd8ed1ab_0
  sqlite                              esri::sqlite-3.41.2-0 --> conda-forge::sqlite-3.43.2-hcfcfb64_0
  vc                          pkgs/main::vc-14.2-h21ff451_1 --> conda-forge::vc-14.3-h64f974e_17
  vs2015_runtime     esri::vs2015_runtime-14.27.29016-h5e5~ --> conda-forge::vs2015_runtime-14.36.32532-h05e6639_17
  wincertstore       pkgs/main/win-64::wincertstore-0.2-py~ --> conda-forge/noarch::wincertstore-0.2-pyhd8ed1ab_1009
  zlib                                  esri::zlib-1.2.13-0 --> conda-forge::zlib-1.2.13-hcfcfb64_5
  zlib-ng                             esri::zlib-ng-2.0.6-1 --> conda-forge::zlib-ng-2.0.7-hcfcfb64_0
  zstd                     pkgs/main::zstd-1.5.2-h19a0ad4_0 --> conda-forge::zstd-1.5.5-h12be248_0

The following packages will be SUPERSEDED by a higher-priority channel:

  blas                                            pkgs/main --> conda-forge
  certifi            pkgs/main/win-64::certifi-2023.7.22-p~ --> conda-forge/noarch::certifi-2023.7.22-pyhd8ed1ab_0
  freetype                          esri::freetype-2.12.1-4 --> conda-forge::freetype-2.12.1-hdaf720e_2
  lerc                   esri/noarch::lerc-4.0-pyh39e3cac_0 --> conda-forge/win-64::lerc-4.0.0-h63175ca_0
  libxml2                  esri::libxml2-2.10.4-arcgispro_0 --> conda-forge::libxml2-2.10.4-hc3477c8_0
  pip                pkgs/main/win-64::pip-23.3-py39haa955~ --> conda-forge/noarch::pip-23.3-pyhd8ed1ab_0
  pycparser          pkgs/main::pycparser-2.21-pyhd3eb1b0_0 --> conda-forge::pycparser-2.21-pyhd8ed1ab_0
  six                                 esri::six-1.16.0-py_0 --> conda-forge::six-1.16.0-pyh6c4a22f_0
  tzdata                 pkgs/main::tzdata-2023c-h04d1e81_0 --> conda-forge::tzdata-2023c-h71feb2d_0
  wheel              pkgs/main/win-64::wheel-0.41.2-py39ha~ --> conda-forge/noarch::wheel-0.41.2-pyhd8ed1ab_0



Downloading and Extracting Packages

blas-1.0             | 1 KB      |            |   0% 
blas-1.0             | 1 KB      | ########## | 100% 
blas-1.0             | 1 KB      | ########## | 100% 

imageio-2.31.5       | 284 KB    |            |   0% 
imageio-2.31.5       | 284 KB    | ########## | 100% 
imageio-2.31.5       | 284 KB    | ########## | 100% 

pip-23.3             | 1.3 MB    |            |   0% 
pip-23.3             | 1.3 MB    | 1          |   1% 
pip-23.3             | 1.3 MB    | ########## | 100% 
pip-23.3             | 1.3 MB    | ########## | 100% 

m2w64-gmp-6.1.0      | 726 KB    |            |   0% 
m2w64-gmp-6.1.0      | 726 KB    | ########## | 100% 
m2w64-gmp-6.1.0      | 726 KB    | ########## | 100% 

openssl-3.1.3        | 7.1 MB    |            |   0% 
openssl-3.1.3        | 7.1 MB    | ###8       |  39% 
openssl-3.1.3        | 7.1 MB    | ########## | 100% 
openssl-3.1.3        | 7.1 MB    | ########## | 100% 

m2w64-gcc-libs-core- | 214 KB    |            |   0% 
m2w64-gcc-libs-core- | 214 KB    | ########## | 100% 
m2w64-gcc-libs-core- | 214 KB    | ########## | 100% 

cffi-1.16.0          | 231 KB    |            |   0% 
cffi-1.16.0          | 231 KB    | ########## | 100% 
cffi-1.16.0          | 231 KB    | ########## | 100% 

cryptography-41.0.4  | 1.0 MB    |            |   0% 
cryptography-41.0.4  | 1.0 MB    | ###9       |  40% 
cryptography-41.0.4  | 1.0 MB    | ########## | 100% 
cryptography-41.0.4  | 1.0 MB    | ########## | 100% 

m2w64-gcc-libs-5.3.0 | 520 KB    |            |   0% 
m2w64-gcc-libs-5.3.0 | 520 KB    | ########## | 100% 
m2w64-gcc-libs-5.3.0 | 520 KB    | ########## | 100% 

m2w64-libwinpthread- | 31 KB     |            |   0% 
m2w64-libwinpthread- | 31 KB     | ########## | 100% 
m2w64-libwinpthread- | 31 KB     | ########## | 100% 

setuptools-68.2.2    | 454 KB    |            |   0% 
setuptools-68.2.2    | 454 KB    | ########## | 100% 
setuptools-68.2.2    | 454 KB    | ########## | 100% 

msys2-conda-epoch-20 | 3 KB      |            |   0% 
msys2-conda-epoch-20 | 3 KB      | ########## | 100% 
msys2-conda-epoch-20 | 3 KB      | ########## | 100% 

sqlite-3.43.2        | 830 KB    |            |   0% 
sqlite-3.43.2        | 830 KB    | ########## | 100% 
sqlite-3.43.2        | 830 KB    | ########## | 100% 

freetype-2.12.1      | 498 KB    |            |   0% 
freetype-2.12.1      | 498 KB    | 3          |   3% 
freetype-2.12.1      | 498 KB    | ########## | 100% 
freetype-2.12.1      | 498 KB    | ########## | 100% 

libsqlite-3.43.2     | 827 KB    |            |   0% 
libsqlite-3.43.2     | 827 KB    | ########## | 100% 
libsqlite-3.43.2     | 827 KB    | ########## | 100% 

m2w64-gcc-libgfortra | 342 KB    |            |   0% 
m2w64-gcc-libgfortra | 342 KB    | ########## | 100% 
m2w64-gcc-libgfortra | 342 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Retrieving notices: ...working... done

Get the data for analysis¶

Search for Multispectral Landsat layer in ArcGIS Online.

In [204]:
landsat_item = gis.content.get('d9b466d6a9e647ce8d1dd5fe12eb434b')
landsat = landsat_item.layers[0]
landsat_item
Out[204]:
Multispectral Landsat
Landsat multispectral and multitemporal imagery with on-the-fly renderings and indices for visualization and analysis. The Landsat 8 and 9 imagery in this layer is updated daily and is directly sourced from the USGS Landsat collection on AWS.Imagery Layer by esri
Last Modified: June 30, 2022
3 comments, 1,028,548 views

Applying Natural color to the filtered Landsat collection using predefined apply function

Select the area of interest¶

In [205]:
# India Reservoirs in Living Atlas
CWC_Reservoirs = "34b71f5ea24b49ce857e8ee5e71a4117"
item = gis.content.get(CWC_Reservoirs)
item
Out[205]:
India: National Register of Large Dams 2019 (Polygon)
This layer shows the information of large dams compiled and maintained by CWC. The nationwide register of large dams i.e. National Register of Large Dams (NRLD) was published in June 2019.Feature Layer Collection by esri_IN_content
Last Modified: August 19, 2022
0 comments, 846 views
In [216]:
%%time

dam_layer = item.layers[0]
dam_layer_filter = dam_layer.query(where="name = 'K.R.Sagara Dam'", as_df=True)
extent = dam_layer_filter.iloc[0]['SHAPE'].extent
extent = {'xmin': extent[0], 'ymin': extent[1], 'xmax': extent[2], 'ymax': extent[3]}
print(extent)

m = gis.map('Mysuru, India')
m.add_layer(dam_layer_filter, options={'opacity':0.1})
m.basemap = 'satellite'
m

{'xmin': 8505585.607700001, 'ymin': 1390822.4349999987, 'xmax': 8524908.163600001, 'ymax': 1408422.3772}
Wall time: 2.18 s

MapView(layout=Layout(height='400px', width='100%'))
In [207]:
rgb_collection = apply(landsat, 'Natural Color with DRA')

Function to create collection of images with desired time intervals¶

The function below creates an array of images with the desired time intervals. If a user specifies 'm' then the images in the selected collection will be consolidated on a monthly basis i.e. all the images of the specified extent will be mosaicked monthly and if the user specifies 'y' as the interval then the images in the selected collection will be consolidated on yearly basis.

In [217]:
from functools import lru_cache

@lru_cache(maxsize=50)
def load_font():
    try:
        if system()=='Windows':
            return ImageFont.truetype("arial.ttf", 30)
        elif system()=='Linux':
            return ImageFont.truetype("~/.fonts/truetype/dejavu/DejaVuSans.ttf", 30)
        else:
            return ImageFont.truetype("Arial.ttf", 30)
    except:
        return ImageFont.load_default()
In [218]:
def collection(df, out_dir, interval, start, end, height, width):
    images=[]
    if(interval=='m'):                                                                                     # monthly
        for i in range(int(start.split('-')[0]), int(end.split('-')[0])+1):
            for j in range(1,13):
                selected = df[(df['AcquisitionDate'].dt.year == i) & (df['AcquisitionDate'].dt.month == j)]
                id = selected['OBJECTID'].values.tolist()
                if(len(id)>0):
                    rgb_collection.mosaic_by(method="LockRaster",lock_rasters=id)
                    img_name = 'img_'+str(i)+"-"+str(j)+".jpg"
                    img_file = os.path.join(out_dir, img_name)
                    rgb_collection.export_image(bbox=extent, size=[height,width], f='image', 
                                                  save_folder=out_dir, 
                                                  save_file=img_name)
                    img = Image.open(img_file).convert('RGB')
                    font = load_font()
                    draw = ImageDraw.Draw(img)
                    draw.text((500, 0),'{:>2}-{:<4}'.format(j,i),(255,255,255),font=font)
                    images.append(img)
                    os.remove(img_file)
                    
    elif(interval=='y'):                                                                                  # yearly
        for i in range(int(start.split('-')[0]), int(end.split('-')[0])+1):
            selected = df[df['AcquisitionDate'].dt.year == i]
            id = selected['OBJECTID'].values.tolist()
            if(len(id)>0):
                rgb_collection.mosaic_by(method="LockRaster",lock_rasters=id)
                img_name = 'img_'+str(i)+".jpg"
                img_file = os.path.join(out_dir, img_name)
                rgb_collection.export_image(bbox=extent, size=[height,width], f='image', 
                                              save_folder=out_dir, 
                                              save_file=img_name)
                img = Image.open(img_file).convert('RGB')
                font = load_font()
                draw = ImageDraw.Draw(img)
                draw.text((500, 0),'{:>4}'.format(i),(255,255,255),font=font)  
                images.append(img)
                os.remove(img_file)
    
    return images

Make video from image collection¶

The function below will generate a movie (gif) from the collection saved from the above step.

In [219]:
def create_movie(target, out_dir, interval, start, end, height, width, extent, duration):
    start_date = datetime.strptime(start, '%Y-%m-%d')
    end_date = datetime.strptime(end, '%Y-%m-%d')
    selected = target.filter_by(where="(Category = 1) AND (CloudCover <=0.5)",
                             time=[start_date, end_date],
                             geometry=arcgis.geometry.filters.intersects(extent))
    df = selected.query(out_fields="AcquisitionDate, GroupName, CloudCover, DayOfYear", 
                        order_by_fields="AcquisitionDate").sdf
    df['OBJECTID'] = df['OBJECTID'].astype(int)
    df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
    frames = collection(df, out_dir, interval, start, end, height, width)
    out_gif = os.path.join(out_dir, 'movie_'+interval+'.gif')
    imageio.mimsave(out_gif, frames, format='GIF', loop=0, duration=duration)
    print("Movie Created")
    return out_gif
In [222]:
%%time

# calling create_movie function for the year 2022 on a monthly basis
#out_gif = create_movie(rgb_collection, output_data_dir, 'm' ,'2022-01-01','2022-12-31', 1000, 1000, extent, 1.0) 

# Call the create_movie function for 1980-2023 for annual interval
out_gif = create_movie(rgb_collection, output_data_dir, 'y' ,'1980-01-01','2023-10-01', 1000, 1000, extent, 1.0)

Movie Created
Wall time: 1min 50s

The movie (gif) will be created in the out_dir directory specified in the create_movie call. The gif below is generated using the code above which shows how the Hirakund reservoir in Odisha, India changed monthly in the year 2019.

In [223]:
display(dImage(data=open(out_gif,'rb').read(), format='png'))

Conclusion¶

The sample notebook shows how you can animate an Imagery Layer over time to get a visual detail of the change that has happened on any given extent, either monthly or annually. You can bring any image collection that you have or use the image service provided by Esri Living Atlas and run this notebook against it. This can be repeated for any other location and for any other time interval by changing the extent variable and the time interval when calling the create_movie() function.

Reset the namespace¶

The following %reset -f command is a built-in command in Jupyter Notebook that will reset the namespace. This is good practice to run when you are finished with the notebook.

In [ ]:
%reset -f